Repository navigation
Conversation
The upcoming fs-verity helper thread receives requests as data over a socket, so it can't be generic over the hash type. Add crate-internal variants of the enable and measure ioctls that take an `Algorithm` value instead. Unlike `fs_ioc_enable_verity::<H>`, the enable variant honours the block size carried by the algorithm. `enable_verity_raw` is unchanged. Generated-by: AI
Relax `#![forbid(unsafe_code)]` to `#![deny(unsafe_code)]`. Unsafe code is still rejected everywhere by default, but a module can now opt in with `#![allow(unsafe_code)]`, which `forbid` doesn't permit. This matches the workspace-wide `unsafe_code = "deny"` lint (composefs#123). The next commit needs this: giving a thread its own file descriptor table takes `unshare(CLONE_FILES)`, which rustix only offers as an unsafe function. Opt-ins can be module-wide or on a single item, and each use must be justified. Generated-by: AI
Linux has no O_CLOFORK, so a writable descriptor that one thread holds can be copied into a child forked by another thread, where it stays open until the child calls exec(). For fs-verity this means FS_IOC_ENABLE_VERITY can fail with ETXTBSY at random. Add a crate-private `isolation` module with spawn_isolated(), which starts a thread that calls unshare(CLONE_FILES) and so gets its own copy of the descriptor table. fork() only copies the calling thread's table, so descriptors opened in the isolated thread (or threads it spawns) can't leak into children forked elsewhere. The given descriptors are moved into the new table, and everything else the thread inherited (other than the standard streams) is closed with close_range(), or via /proc/thread-self/fd on kernels older than 5.9. Otherwise the thread would keep the rest of the process's pipes and sockets open. rustix doesn't provide close_range(), so this adds a libc dependency for now. spawn_isolated() is unsafe: descriptor numbers mean different things in the two tables, and the compiler can't stop the callback from reaching descriptors through process-wide state such as a logger. To limit mistakes the callback is a plain fn pointer taking a Copy argument, so it can't capture descriptors. This module holds the unsafe code; callers need a local `#[allow(unsafe_code)]` to call it. Generated-by: AI
Enabling fs-verity requires that no writable descriptor to the file exists. If another thread forks while we hold the descriptor used to fill a new file, the child inherits it, and enabling verity fails with ETXTBSY until the child calls exec(). The current workarounds retry a fixed number of times (enable_verity_with_retry) or enable verity on a copy that the caller can easily ignore (enable_verity_maybe_copy). Add FsVerityHelper, which does all the work on threads with a private descriptor table (via crate::isolation), so the writable descriptor never exists in the table fork() copies: - create() returns a VerityWriter that feeds the file through a pipe. It implements Write and tokio's AsyncWrite, and pipe_mut() exposes the pipe so std::io::copy() can use splice(). finish() commits, and dropping the writer discards the file. - create_from_fd() copies an existing file, by reflink if possible. - finish() returns a VerityFile with the kernel-measured digest, or FinishError::NotSupported(UnverifiedFile) if the filesystem lacks fs-verity support, so callers can't mistake one for the other. - publish() links the result into place, by linkat() for O_TMPFILE files and by renameat(RENAME_NOREPLACE) for the named fallback used on filesystems without O_TMPFILE. Temporary names are cleaned up on every failure path. - wait() and shutdown() give an explicit graceful shutdown; dropping the last handle tears the threads down too. Requests and results travel over SOCK_SEQPACKET sockets with SCM_RIGHTS. Since our ends of the pipes and sockets can still be inherited by forked children, end of data and shutdown are signalled explicitly instead of by waiting for the descriptors to close. The helper uses the runtime-Algorithm ioctl wrappers added earlier, so their dead_code expectations are removed. Nothing uses the helper yet. Generated-by: AI
Two additions to FsVerityHelper for the repository code to come: - FsVerityHelperBuilder::userspace_digest(true) makes the helper compute the digest in userspace on filesystems without fs-verity, while it copies the data, and return it with the UnverifiedFile. Insecure repositories need the digest to name the object, and this keeps them at a single pass over the data. To avoid hashing files the kernel will hash anyway, the helper finds out once per filesystem whether fs-verity works there by trying it on an empty file. Off by default, so callers that don't need the digest don't pay for it. - reflink_from_fd() is create_from_fd() without the copy fallback. It fails with FinishError::ReflinkUnsupported, so that the caller can try something else first: Repository::ensure_object_from_file() tries a hardlink before copying. Both VerityFile and UnverifiedFile now also report how their data got there (CopyMethod), so the repository can tell a reflink from a copy. Generated-by: AI
Every object (and a strict meta.json) used to be written through a writable descriptor in the process's shared table and then given fs-verity with enable_verity_maybe_copy(). If another thread forked in the meantime, verity went onto a copy, which callers had to remember to use. finalize_object_tmpfile_impl() didn't, for the named-tmpfile fallback: it renamed the original, verity-less file into place, so a secure repository could end up with an object without verity. Create all object files through an FsVerityHelper owned by the repository (started on first use, one worker per allowed concurrent write), so the writable descriptor never exists where fork() can copy it: - Repository::create_object(size) returns an ObjectWriter (Write, AsyncWrite, pipe_mut(), finish(), finish_async()). It replaces the public create_object_tmpfile()/finalize_object_tmpfile() pair. - ensure_object(), ensure_object_from_reader() and ensure_object_from_fd() go through the helper; the latter copies with reflink or copy_file_range() inside the helper. - ensure_object_from_file() keeps its reflink -> hardlink -> copy order and ImportContext cache, using the helper's reflink-only mode. - A strict meta.json is written by a one-shot helper and fsynced. - Insecure repositories name objects by the digest the helper computes in userspace where the filesystem lacks fs-verity, still in one pass. - The named-tmpfile fallback, its cleanup and dedup now live in the helper and VerityFile::publish(), replacing ObjectTmpfile, NamedTmpfileGuard, link_tmpfile_as_object() and friends. Two places enable verity on files we don't own and now use enable_verity_raw() without retrying: try_hardlink_object() (a busy source file now fails the hardlink, so the import copies instead, or fails in zerocopy mode) and re-enabling verity on an existing object in an insecure repository (a busy object is left as it is, after checking its content). composefs-oci moves to create_object(), and to the new create_scratch_tmpfile() for scratch files that never become objects. store_blob_async() now takes the blob size, which the helper checks. A new test stores objects in a secure repository with the named-tmpfile fallback while other threads fork, and checks that each one has verity. Against the old code it fails about two runs in three. Generated-by: AI
FlatDigestStore (used by `mkcomposefs --digest-store`) copied each file through a writable descriptor in the shared table and then enabled fs-verity with enable_verity_maybe_copy(), with the same exposure to forked children as the repository had. Give it an FsVerityHelper, sized by its concurrency limit and computing userspace digests in insecure mode, and create objects with create_from_fd(). As a bonus the data is now reflinked where possible, and filesystems without O_TMPFILE get the named-tmpfile fallback. The repository's private enum for "a file with or without verity" moves next to the helper as FinishedFile so both can use it, and the now unused reopen_tmpfile_ro() and create_tmpfile_in() are removed. Generated-by: AI
…copy Nothing in the tree uses them any more. Retrying a fixed number of times can't guarantee success, and enable_verity_maybe_copy() makes it easy to keep using the original, verity-less file when it returns a copy. Point users at FsVerityHelper, which avoids the problem. enable_verity_raw() stays as it is. Generated-by: AI
|
Unreviewed but I did a lot of hand-holding to try and keep it on-track. I am generally encouraged by the fact that we ended up with net-negative code in the existing repository/fs. We had a boatload of closely-related duplicate code there that was doing similar stuff but doing it inconsistently or outright wrong in some cases. Big-picture this seems to resolve most if not all of that. |
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent imports can deadlock, and worker panics can execute application hooks against the helper’s private descriptor table, risking data corruption and violating its safety contract.
5 open issues
Use async stream writes to avoid deadlocks on current-thread runtimes · New Prevent panic hooks from accessing the private descriptor table · New Report worker spawn failures instead of queuing unserviceable requests · New Explicitly close the reply socket when a worker panics · New Validate block-size exponents before shifting · New
What changed in this PR
This PR addresses #442 by moving fs-verity file creation into an isolated helper, preventing writable descriptors from leaking into forked children and making verified results explicit in the API.
Changes:
- Adds isolated worker threads, streaming writers, and typed verified/unverified results.
- Migrates repository and OCI ingestion to the helper, centralizing publication and temporary-file cleanup.
- Adds regression tests and deprecates retry/copy-based verity enablement APIs.
| File | Description |
|---|---|
crates/composefs/src/util.rs |
Removes superseded temporary-file helpers. |
crates/composefs/src/repository.rs |
Uses helper-backed object writers and adds regression tests. |
crates/composefs/src/lib.rs |
Registers isolation module and permits explicit unsafe opt-ins. |
crates/composefs/src/isolation.rs |
Adds threads with private descriptor tables. |
crates/composefs/src/fsverity/mod.rs |
Exports helper APIs and deprecates older workarounds. |
crates/composefs/src/fsverity/ioctl.rs |
Adds runtime-algorithm ioctl wrappers and tests. |
crates/composefs/src/fsverity/helper.rs |
Implements isolated creation, copying, hashing, and publication. |
crates/composefs/src/fs.rs |
Migrates FlatDigestStore to the helper. |
crates/composefs/Cargo.toml |
Adds dependencies and features needed by isolation and pipe I/O. |
crates/composefs-oci/src/tar.rs |
Streams tar objects through the new writer API. |
crates/composefs-oci/src/skopeo.rs |
Supplies blob sizes and uses scratch temporary files. |
crates/composefs-oci/src/oci_layout.rs |
Supplies expected sizes for artifact blob storage. |
crates/composefs-oci/src/layer.rs |
Uses asynchronous object writers with explicit blob sizes. |
crates/composefs-oci/src/delta.rs |
Uses scratch temporary files for reconstruction. |
crates/composefs-integration-tests/src/tests/privileged.rs |
Updates a comment describing the ingestion path. |
Review effort: Balanced
| // Run the progress driver concurrently with the blob store. | ||
| let (store_result, ()) = tokio::join!(store_blob_async(repo, async_file), progress_driver); | ||
| let (store_result, ()) = tokio::join!( | ||
| store_blob_async(repo, async_file, layer_size), |
| let control = unsafe { | ||
| crate::isolation::spawn_isolated( | ||
| "fsverity-helper".into(), | ||
| vec![theirs], | ||
| self.max_workers, |
| Err(e) if workers.is_empty() => { | ||
| send_error(&job.reply, "starting fs-verity worker", &e); | ||
| continue; | ||
| } | ||
| // Queue it for the workers we have. | ||
| Err(_) => {} |
| // A panic drops the job, and with it the reply socket, so | ||
| // the caller gets an error; the worker carries on. | ||
| let _ = std::panic::catch_unwind(AssertUnwindSafe(|| run_job(job, probes))); |
| composefs_ioctls::fsverity::fs_ioc_enable_verity( | ||
| fd.as_fd(), | ||
| algorithm.kernel_id(), | ||
| 1u32 << algorithm.lg_blocksize(), |
|
Ran through bootc CI over at bootc-dev/bootc#2637 and everything passes so that's reassuring Still need to give the code some eyeball review |
| not(test), | ||
| expect(dead_code, reason = "used by FsVerityHelper in the next commit") | ||
| )] | ||
| pub(crate) unsafe fn spawn_isolated<A: Copy + Send + 'static>( |
There was a problem hiding this comment.
This requires careful review. I feel like it'd be more maintainable to use e.g. https://docs.rs/procspawn/latest/procspawn/ - this is the crate I was talking about in the community meeting on Friday that's a bit like Python's multiprocessing.
The spawned helper should be single threaded at least by default.
The important thing here is we should always go through a fork+exec() to ensure we start from a clean slate. A raw fork() or unshare() is much more dangerous.
|
Closing in favor of #448 |


Closes #442 (design discussion in #442 (comment)).
Problem
fs-verity can only be enabled when no writable descriptor to the file exists. If another thread forks while we hold the descriptor used to fill a new file, the child inherits it, and
FS_IOC_ENABLE_VERITYfails withETXTBSYuntil the child execs. Linux has noO_CLOFORK.The current workarounds are
enable_verity_with_retry(three tries, 1ms apart, no guarantee) andenable_verity_maybe_copy(returns anOption<OwnedFd>copy that is easy to ignore). The tree had a real instance of that misuse: on the named-tmpfile fallback path,finalize_object_tmpfile_implrenamed the original, verity-less file into place, so a secure repository could store an object without verity.Approach
A helper thread calls
unshare(CLONE_FILES)to get a private file descriptor table.fork()only copies the calling thread's table, so writable descriptors to new files, created by the helper and its worker threads, can never leak into children forked elsewhere. Callers feed data in through a pipe or hand over a source fd, and get back a read-only fd with verity already enabled. Fds cross between the tables only viaSCM_RIGHTS.Commits
Algorithmenable/measure. Internal ioctl wrappers; the helper takes requests as data, so it can't be generic over the hash type.forbid(unsafe_code)→deny. A separate commit so it stands out: unsafe is still rejected by default, but a module can now opt in.spawn_isolated(). The only new unsafe code:unshare(CLONE_FILES)plus closing inherited fds (close_rangevialibc, since rustix doesn't expose it yet).FsVerityHelper.create()returns aVerityWriter(Write,AsyncWrite,pipe_mut()for splice).create_from_fd()copies an existing file.finish()returnsVerityFile, orFinishError::NotSupported(UnverifiedFile), so "not enabled" can't be mistaken for "enabled".publish()links the result into place (linkatfor O_TMPFILE files,RENAME_NOREPLACEfor the named fallback).wait()/shutdown()are explicit; dropping the last handle tears the threads down.userspace_digest(true): on filesystems without verity, hash during the single copy, checked once per filesystem.reflink_from_fd()andCopyMethod.Repository::create_object(size)→ObjectWriterreplaces the publiccreate_object_tmpfile/finalize_object_tmpfile. All object paths and strictmeta.jsongo through the helper.publish().store_blob_asyncnow takes the blob size.FlatDigestStorethrough the helper. It now also gets reflinks and the O_TMPFILE fallback.enable_verity_with_retryandenable_verity_maybe_copy.enable_verity_rawis unchanged.Notes for reviewers
create_object_tmpfile/finalize_object_tmpfileare removed (usecreate_object/create_scratch_tmpfile), andstore_blob_asyncgains asizeparameter.repository.rs,fs.rsandutil.rsare net negative (about −190 lines of code). The new helper and isolation code is about +1,200.ensure_objectcalls are about 25 µs slower (52 → 77 µs), from the helper round trip. Sending small payloads inline in the request message is a possible follow-up.try_hardlink_objectand re-enabling verity on an existing object (insecure repos) useenable_verity_raw.write_concurrencyon one thread deadlocks. This is documented, and existing callers are bounded bywrite_semaphore.CFS_TEST_TMPDIRon ext4 (needs repository: Explicitly unlock image in test_gc_keeps_locked_image #446 fortest_gc_keeps_locked_image).Generated-by: AI
The design was iterated with an AI agent; the code is LLM-generated and has not yet been fully reviewed by me.